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EXECUTIVE  SUMMARY 


Overview 

Due  to  numerous  constraints  (e.g.,  airspace,  cost,  safety),  a  simulated  environment  is 
currently  the  most  feasible  venue  for  US  Air  Force  fighter  pilots  to  practice  in  a  four  versus 
many  opponents  engagement.  It  is  also  one  of  the  few  opportunities  these  pilots  have  to  work  as 
a  team  with  E-3  airborne  warning  and  control  system  (AW ACS)  controllers.  Logistically,  it  is 
almost  impossible  to  assemble  the  resources  necessary  to  conduct  these  types  of  exercises  in  real 
life  (e.g.,  airspace  for  20+  aircraft,  maintenance,  etc.).  The  Operational  Requirements  Document 
(Hawley,  1997)  outlines  the  requirements  for  a  simulation  that  is  as  real  as  possible.  As  a  result, 
improved  visual  displays  may  be  necessary  to  further  enhance  the  realism  and  thus  transfer  of 
training  potential  for  the  simulated  flights. 

At  the  request  of  the  Air  Force  Research  Laboratory,  Human  Systems  Information 
Analysis  Center  (HSIAC)  performed  the  first  three  steps  of  an  established  eight  step  process 
designed  to  assess  the  costs  and  relative  benefits  of  selected  alternatives  to  one  technology  area, 
the  visual  display  system,  of  the  distributed  mission  training-aircrew  (DMT-A)  program.  Two 
investment  alternatives  were  selected  with  respect  to  the  visual  display  system.  The  choices 
were  to  continue  using  the  existing  display  system  or  to  upgrade  to  an  enhanced  20/20  Immersive 
Visual  Display  System.  This  report  will  begin  the  process  that  Air  Force  Research  Laboratory 
(AFRL)  can  use  to  decide  which  investment  alternative  has  the  greatest  payoff  with  respect  to  the 
established  goals  of  the  DMT-A  program. 

Study  Methodology 

Assessment  of  Current  Environment.  The  assessment  of  the  current  environment  was 
a  two-part  process.  First,  Human  Systems  LAC  performed  an  in-depth  literature  search  on 
relevant  and  reasonable  resources  relating  to  distributed  mission  training  (DMT)  generally  and 
the  visual  display  system  specifically.  Second,  a  Human  Systems  IAC  analyst  visited  the  DMT 
facility  in  Mesa,  Arizona  to  witness  the  existing  system  in  use  first  hand.  This  real-life  look  at 
the  capabilities  and  shortfalls  of  the  current  visual  system  provided  invaluable  insight  into  the 
DMT  environment  as  it  currently  exists.  Additional  information  provided  by  the  sponsor  was 
also  used  to  further  supplement  the  information  gathered  by  Human  Systems  IAC. 

This  assessment  found  that  the  current  display  system  is  relatively  effective  in  meeting 
its  training  mission.  However,  the  displays  have  some  deficiencies  that  could  be  improved  upon 
(e.g.,  resolution,  brightness). 

Gap  Analysis.  The  gap  analysis  concentrated  on  comparing  the  goals  and  needs  of  the 
Air  Force  versus  the  areas  where  those  needs  are  and  are  not  being  met  by  the  existing  visual 
display  system.  The  result  was  identification  of  three  key  areas  that  require  the  most  work  in 
reaching  stated  goals: 

•  Improvement  of  simulator  and  sensor  image  resolution. 

•  Inclusion  of  technology  upgrades: 

•  Collimated  projection  screens, 

•  High  resolution,  full  color  laser  projectors,  and 

•  High  performance,  low-cost  image  generators. 

•  Reduction  of  display  system  cost. 
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Identification  of  Investment  Alternatives.  The  identification  of  investment 
alternatives  was  accomplished  through  the  comments  and  suggestions  of  subject  matter  experts 
and  the  results  of  the  in-depth  research  review. 

Based  on  the  information  available,  the  human  effectiveness  directorate  of  AFRL 
(AFRL/HE)  has  two  choices  with  respect  to  visual  displays  systems  for  DMT-A.  AFRL/HE  can 
either  keep  the  current  displays  and  supporting  equipment  or  elect  to  invest  in  some  version  of 
the  20/20  Immersive  Visual  Display  System. 

Conclusions  and  Recommendations 

This  study  provides  a  start  to  the  process  needed  to  accurately  estimate  the  costs  and 
benefits  associated  with  two  potential  visual  display  alternatives.  It  also  illustrates  the  process 
for  the  remaining  steps  to  complete  the  entire  analysis. 

Based  on  the  information  available,  the  DMT-A  program  is  an  effective  tool  for  training 
pilots  to  work  as  a  team.  The  gap  analysis  illustrates  that  the  visual  display  resolution 
requirements  are  not  being  met  and  the  visual  display  system  needs  to  be  upgraded.  A  potential 
upgrade  to  the  status  quo  visual  display  system  is  the  20/20  Immersive  Display  System.  The  next 
steps  to  determining  whether  a  new  display  system  is  warranted  would  be  to  assess  the  costs  and 
benefits  associated  with  each  display  system.  In  addition,  an  evaluation  of  sensitivity  and  risk 
would  need  to  be  accomplished.  Following  that,  the  net  value  analysis  of  the  investment 
alternatives  would  determine  which  was  the  preferred  alternative. 
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1.  INTRODUCTION 


1.1  BACKGROUND 

Success  in  modem  aerospace  warfare  is  dependent  on  the  forces  working  well  together. 
For  this  coordination  to  be  present  prior  to  an  actual  conflict,  it  is  important  for  the  forces  to  train 
together  as  they  intend  to  fight,  as  an  effective  team  (MNS,  2000).  To  this  end,  the  Mission 
Need  Statement  (MNS)  for  the  Air  Force  National  Air  and  Space  (Warfare)  Model,  identified  the 
need  to  provide  a  training  environment  which  integrates  all  Air  Force  missions.  The  training 
program  should  also  be  capable  of  training  warfighters  at  all  echelons  from  the  Joint  Tactical 
Force  (JTF)  Commander  down  to  the  individual  level.  According  to  the  MNS,  high-fidelity 
simulation,  when  coupled  with  data  collection,  mission  replay,  and  performance  feedback  and 
evaluation,  can  improve  the  realism  and  value  of  training,  enhance  day-to-day  readiness,  and 
provide  opportunities  to  test  new  tactics,  techniques,  and  procedures  (Hawley,  1999). 

Prior  to  distributed  mission  training  (DMT),  there  were  several  limitations  present  in 
existing  training  options  that  made  it  difficult  to  conduct  effective  training.  From  these 
difficulties,  DMT  was  bom.  For  instance,  live-fly  training  exercises  are  cost  prohibitive,  present 
environmental  and  safety  concerns,  and  are  severely  limited  due  to  airspace  restrictions.  While 
effective  at  training  individual  skills,  single-ship  simulators  were  unable  to  allow  operators  to 
train  as  crews  or  as  teams.  Pilots  for  aircraft  types  tended  to  train  separately,  focusing  only  on 
individual  skills.  A  final  limitation  to  the  existing  training  system  that  helped  lead  to  the 
development  of  DMT  was  the  limitation  to  sensor,  communications,  and  navigation  data.  Air 
Force  Space  Command  forces  could  not  train  or  rehearse  at  full  capabilities  at  the  same  time  as 
other  air  or  ground  components.  Additionally,  the  lack  of  functionality  and  realism  in  existing 
simulators  made  DMT  necessary  (Hawley,  1997). 

"The  primary  purpose  of  DMT  is  to  train  aerospace  crews  and  system  operators  in  the 
individual,  team,  and  composite  force  skills  needed  to  accomplish  their  missions"  (Hawley, 

1998).  Put  simply,  DMT  was  designed  to  help  Air  Force  operators  train  the  way  they  will  be 
expected  to  fight.  DMT  was  developed  to  fulfill  the  need  to  provide  a  training  environment  that 
could  integrate  Air  Force  missions  at  all  levels  of  experience,  from  the  JTF  Commander  down  to 
the  individual  (agent)  level  (Hawley,  1998).  Its  intent  was  to  improve  the  realism  and  value  of 
training,  enhance  day-to-day  readiness,  and  provide  opportunities  to  test  new  tactics,  techniques, 
and  procedures  through  enhanced,  high-fidelity  simulation. 

DMT  is  an  Air  Force  readiness  tool  that  utilizes  emerging  technologies  to  improve  and 
integrate  training  and  mission  rehearsal.  DMT  is  not  a  single  project,  but  a  federation  of  efforts 
organized  into  five  primary  domains  (e.g.,  aircrew,  mobility,  special  ops,  space,  and  command 
and  control).  Each  domain  has  its  own  proponents,  users,  and  focus  (Lollar,  2000).  This  report 
is  in  support  of  that  portion  of  DMT  that  supports  aircrew  training  and  readiness,  DMT-A. 

DMT-A  is  an  Air  Combat  Command  (ACC)  led  program  that  focuses  on  the  readiness 
needs  of  combat  aircrews.  A  primary  goal  of  the  DMT-A  program  is  to  effectively  train 
warfighters  to  fight  as  a  team.  The  primary  warfighters  using  DMT-A  are:  (Lollar,  2000). 

•  Pilots,  weapons  systems  officers,  and  navigators  of  primary  combat  aircraft  (e.g.,  F- 
15C,  F-16C,  B-l) 

•  Weapons  controllers  and  directors  of  combat  support  aircraft  (e.g.,  AW  ACS, 
JSTARS,  Rivet  Joint) 

•  Weapons  controllers  assigned  to  Air  Control  Squadrons  based  with  fighter  wings 
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In  addition  to  training  warfighters,  DMT-A  also  supports  seven  core  Air  Force  tasks  and 
associated  subtasks  (Lollar,  2000): 

•  Air  and  Space  Superiority, 

•  Precision  Engagement, 

•  Information  Superiority, 

•  Global  Attack, 

•  Agile  Combat  Support, 

•  Global  Mobility,  and 

•  Command  and  Control. 


DMT-A  uses  training  and  rehearsal  as  the  sources  for  developing  performance  measures 
to  determine  success  in  each  of  the  essential  areas  of  competency  for  DMT-A  listed  in  Table  1 . 

Table  1.  DMT-A  Mission  Essential  Competencies 


Level 

Training 

Rehearsal 

Individual 

Weapons  employment 

Threat  reactions 

Mission  qualification 

Fighter  weapons  instructor  course  preparation 

Theater  area 
familiarization 

Team 

Mission  preparation,  briefing  and  debriefing 
Rapid  scenario  development 

Flight  leader  upgrade 

Rules  of  engagement 

Target  area  de-confliction 

Interteam 

Large  force  employment 

Mission  commander  upgrade 

Battle  management 

Mission  performance 
analysis 

Combat  mission  practice 

(from  Lollar,  2000) 


The  ability  to  train  as  they  will  fight  improves  the  readiness  of  fighter  and  AWACS 
aircrew  and  allows  for  practice  opportunities  not  available  in  live  flight.  To  this  end,  it  has  been 
established  that  more  realistic  representation  of  the  synthetic  environment  will  improve  transfer 
of  training  and  reduce  negative  training  while  in  the  simulator.  That  is,  operators  can  easily 
apply  skills  learned  in  the  simulator  to  live-flight  and  that  no  bad  habits  are  learned  while 
practicing  in  simulation  (Hawley,  1997). 

The  focus  of  this  cost  benefit  analysis  (CBA)  is  on  the  display  portion  of  the  aircrew  flight 
simulators.  Described  fully  in  Section  2.1.2  of  this  report,  these  displays  include  360  degree 
field  of  view  (FOV),  limited  air-to-air  and  air-to-ground  representation,  and  contribute  to  a 
reasonably  effective  and  realistic  training  environment. 

1.2  PURPOSE  OF  CBA 

In  general,  a  CBA  is  a  systematic  method  of  assessing  the  economic  desirability  of 
selected  investment  alternatives  based  on  their  respective  costs  and  benefits.  The  objective  of 
the  assessment  is  to  identify  one  alternative  that  is  preferable  to  all  of  the  others. 

The  focus  of  this  CBA  is  to  evaluate  the  economic  desirability  of  two  potential  DMT-A 
display  systems.  The  alternative  eventually  implemented  should  result  in  an  improvement  in 
simulated  visual  and  sensor  image  resolution.  It  should  consist  of  low-cost  image  generators, 
laser  projectors,  and  projection  screens.  In  addition,  an  overall  reduction  in  the  cost  of  the 
display  system  is  desired. 
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1.3  OVERVIEW  OF  APPROACH 


As  stated  above,  a  CBA  is  a  systematic  method  employed  to  make  rational  decisions 
between  alternative  systems  by  comparing  their  estimated  costs  and  evaluated  benefits. 
Conducting  a  CBA  assists  in  the  allocation  of  limited  resources  by  probing  each  investment 
alternative  so  that  all  questions  relevant  to  an  appropriate  decision  are  answered.  In  a  complete 
CBA,  the  overall  objective  and  alternatives  are  clearly  defined;  costs  and  benefits  are  fully 
presented;  and  important  assumptions,  factors  and  judgments  are  highlighted.  Outlined  in  the 
following  sections,  there  are  eight  steps  used  by  Human  Systems  IAC  in  conducting  a  CBA. 

1 .  Assess  Current  Environment, 

2.  Perform  Gap  Analysis, 

3.  Identify  Investment  Alternatives, 

4.  Estimate  Cost, 

5.  Perform  Sensitivity  Analysis, 

6.  Characterize  and  Value  Benefits, 

7.  Determine  Net  Value  of  Each  Alternative,  and 

8.  Perform  Risk  Analysis. 

Due  to  resource  limitations  and  specified  customer  needs,  this  CBA  will  focus  on  a 
single  technology  (visual  display  system)  within  the  larger  gamut  of  the  entire  DMT-A  program 
and  perform  the  first  three  steps  of  the  eight  step  process.  The  report  also  sets  the  stage  for 
conducting  a  complete  CBA  on  that  system  by  outlining  the  remaining  five  necessary  steps  to 
complete  the  analysis  on  the  selected  system.. 

1.4  GLOBAL  GROUND  RULES  AND  ASSUMPTIONS 

The  following  ground  rules  and  assumptions  were  used  in  developing  a  plan  to  perform  a 
CBA  on  DMT-A  visual  display  systems: 

•  Two  DMT-A  visual  display  investment  alternatives  will  be  identified. 

•  Three  of  the  eight  CBA  steps  will  be  completed  and  documented. 

•  The  process  for  the  last  five  CBA  steps  will  be  provided. 

•  Alternatives  will  be  compared  to  the  target  goals  established  in  the  gap  analysis. 

•  Technology  or  programs  peripheral  to  the  agreed-upon  visual  display  system  will  not 
be  addressed. 
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2.  COST  BENEFIT  ANALYSIS 


2.1  CURRENT  ENVIRONMENT 

2.1.1  Process 

In  evaluating  the  current  environment,  Human  Systems  IAC  investigated  the  types  of 
technologies  currently  in  use.  Once  familiar  with  the  overall  system,  the  need  for  a  material 
solution  based  on  the  objectives  was  considered. 

The  current  technology  in  distributed  aircrew  training  was  surveyed  and  thoroughly 
examined  to  establish  a  baseline  for  the  analysis.  At  the  same  time,  the  scope  of  the  system  was 
established  through  collaboration  between  Human  Systems  IAC  and  the  sponsor.  That  is,  the 
lines  between  what  would  and  would  not  be  included  in  the  CBA  were  drawn.  For  example,  with 
DMT-A,  connectivity  between  various  simulators  is  a  significant  issue.  While  the  ability  to 
communicate  with  other  systems  is  an  important  aspect  of  DMT-A,  it  does  not  influence  the 
display  systems  and  will  not  be  included  within  the  scope  of  this  study. 

In  order  to  properly  assess  the  current  environment,  Human  Systems  IAC  analysts  first 
"got  smart"  on  the  topic  at  hand.  This  began  with  an  in-house  literature  search  to  ascertain  the 
general  nature  of  the  topic.  Based  on  this  search,  a  comprehensive  list  of  keywords,  catch 
phrases,  subject  matter  experts  (SMEs),  and  example  articles  was  compiled.  This  list  was 
refined  and  combined  with  a  short  background  description  of  the  topic,  the  problem,  and  the 
expected  outcome.  As  is  often  the  case,  the  customer  is  a  SME  in  the  field  and  knows  a  great 
deal  about  the  specific  topic  of  visual  display  training  systems.  Therefore,  this  search  strategy 
was  developed  in  close  coordination  with  the  customer  in  the  form  of  review  and  feedback. 

Once  coordination  was  received,  the  search  strategy  was  provided  to  a  Human  Systems  IAC 
search  specialist  who  conducted  an  in-depth  literature  review  of  all  appropriate  databases  (See 
Appendix  A  for  Search  Strategy).  In  the  case  of  DMT,  the  search  resulted  in  the  identification  of 
literally  hundreds  of  citations.  These  citations  were  surveyed  for  the  most  relevant  and  insightful 
resources.  The  emerging  sources  became  the  backbone  literature  source  of  information  used 
throughout  this  CBA. 

Once  the  literature  search  was  completed,  selected  documents  were  acquired.  The 
documented  information  was  reviewed  and  summarized  to  provide  an  accurate  perspective  of  the 
current  system.  In  the  case  of  DMT-A,  it  was  established  that  the  current  technology  provides 
limited  visual  display  capability.  Further  results  of  this  assessment  are  described  in  the  following 
section. 

2.1.2  Analysis 

According  to  the  DMT  Operational  Requirements  Document  (ORD)  (Hawley,  1997), 
realism  is  essential  to  effective  training.  Therefore,  a  key  performance  parameter  (KPP)  in  aerial 
combat  simulators  is  effective  visual  displays.  Parameter  li  of  the  ORD  states,  "The  key  to 
training  in  the  synthetic  simulation  environment  is  realism.  Visual  realism  is  a  key  factor  when 
trying  to  provide  realistic  training  for  combat  employment.  Failure  to  achieve  high  fidelity  visual 
cueing  will  result  in  negative  training  and  impact  combat  capability."  It  is  clear  from  the 
program  requirements  that  the  visual  display  system  used  in  DMT-A  training  simulators  must  be 
as  realistic  as  possible. 
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Central  components  of  this  effort  are  unit-level  mission  training  centers  (MTCs)  located 
on  individual  Air  Force  bases  (Hawley,  1998).  MTCs  are  made  up  of  four  functional  flight 
simulators  (e.g.,  four  F-16  or  F-15  simulators)  with  an  AW  ACS  station,  mission  control  stations, 
and  interactive  debrief  capabilities.  Each  simulator  consists  of  a  fully-functional  aircraft  cockpit 
and  an  immersive  360  degree  field  of  view  display  (see  Figure  1).  While  in  simulated  flight,  the 
operator  experiences  a  very  realistic  representation  of  his/her  aircraft.  The  mission  control 
stations  include  several  large  displays  that  provide  valuable  information  to  instructors  or 
evaluators  (e.g.,  radar  information,  out-the-window  visuals).  The  MTC  is  supported  by  a  local 
AW  ACS  station  that  allows  a  controller  to  practice  his/her  skills  simultaneously  with  the  fighter 
pilots  and  improve  the  overall  teamwork  needed  in  combat  situations.  This  collection  of 
capabilities  allows  pilots  to  brief,  fly  together,  and  debrief  in  a  few  hours — fall  within  the  same 
facility. 


Figure  1.  Simulator  Graphic  (from  Quad-chart) 

The  current  DMT  visual  display  system,  while  effective  in  most  aspects  of  simulation 
training,  is  somewhat  limited  in  its  capabilities.  For  example,  its  limited  display  resolution 
restricts  the  number  of  targets  displayed  at  one  time.  The  lack  of  resolution  also  inhibits  the 
operator's  perception  of  an  object's  size  and  distance  within  the  visual  scene.  With  the  current 
display  system,  it  is  difficult  for  aviators  to  determine  the  type  (e.g.,  F-16  or  SU-37)  and  distance 
of  another  aircraft.  Far  off  aircraft  are  simply  "dots"  on  the  projected  display.  To  maintain  a 
semblance  of  reality,  the  "dots"  are  color-coded  depending  on  whether  they  are  friend  or  foe 
(e.g.,  green  for  friend  and  red  for  foe).  This  color  coding  attempts  to  provide  the  pilots  the  same 
visual  identification  capabilities  present  in  live  flight.  A  higher  resolution  display  system  with  an 
increased  number  of  pixels  could  provide  improved  realism. 
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Another  limitation  of  the  current  display  system  is  the  brightness  of  the  image.  The 
current  image  generators  simply  are  not  bright  enough  to  provide  sharp,  realistic  depictions  of 
simulated  reality.  Improved  image  generators  could  alleviate  this  problem. 

A  final  problem  with  the  current  display  system  that  could  also  be  eliminated  with 
improved  imaging  is  ground  realism.  As  it  stands,  ground  images  are  poorly  represented  and 
unrealistic,  specifically  when  it  comes  to  targeting  objects.  Currently,  computer  generated 
ground  targets  stand  out  sharply  from  the  basic  background  and  make  it  overly  easy  for  pilots  to 
locate,  to  target,  and  to  destroy.  The  outcome  is  that  pilots  are  not  being  sufficiently  challenged 
as  they  would  be  in  live  flight.  This  could  yield  negative  training,  which  is  specifically  contrary 
to  the  mission  of  DMT-A.  Improving  the  visual  display  system  would  eliminate  this  detrimental 
issue. 

2.2  GAP  ANALYSIS 

2.2.1  Process 

This  step  of  the  CBA  investigates  the  gap  between  the  current  technology  in  use  and  the 
desired  state.  By  understanding  the  requirements  of  the  desired  system  and  the  effectiveness  of 
the  current  technology  (baseline)  in  meeting  those  requirements,  it  is  determined  where  mission 
needs  are  not  being  met.  In  addressing  the  gap,  it  was  the  task  of  Human  Systems  IAC  to 
identify  critical  shortfalls  in  DMT  image  generation  that  influence  training.  For  example,  live- 
flight  exercises  in  modem  fighter  aircraft  provide  effective  practice  at  formation  flying,  while 
they  are  not  as  effective  in  training  against  multiple  adversaries.  On  the  other  hand,  simulators 
can  represent  several  opponents,  but  limitations  in  image  generation  make  it  impossible  to 
practice  formation  flying  in  simulation.  As  a  result,  there  is  a  training  gap  that  exists  in  the 
simulator  environment  due  to  visual  display  limitations.  Described  in  Section  2. 1 .2,  certain 
aspects  of  live  flight  are  not  represented  well  in  current  DMT-A  simulation  (e.g.,  distance/size 
cues,  ground  target  realism).  It  is  this  gap  that  should  be  addressed  by  alternative  system(s).  In 
the  case  of  DMT-A,  the  investment  altemative(s)  are  intended  to  reduce  or  eliminate  the  training 
gap  so  that  operators  can  develop  exceptional  piloting  skills  without  also  picking  up  negative 
habits. 

2.2.2  Analysis 

Stated  in  the  ORD  (Hawley,  1997),  the  visual  display  systems  for  DMT-A  should  be  as 
realistic  as  possible.  This  environment  should  include  terrain,  the  atmosphere,  and  atmospheric 
effects.  Hawley  (1998)  states  that  the  visual  display  system  should,  "Allow  seamless  integration 
of  live  systems  and  events  with  virtual  and  constructive  simulations  and  environments" 
(paragraph  4.3.1).  The  current  system  does  not  do  this.  As  a  result,  an  alternative  system  that 
can  provide  a  seamless  transfer  of  training  from  simulator  to  aircraft  is  needed. 

The  selected  display  alternative  should  result  in  improved  simulated  visual  and  sensor 
image  resolution.  This  improved  resolution,  the  result  of  an  increased  number  of  display  pixels, 
is  intended  to  enhance  display  realism  and  allow  for  better  judgment  of  target  size  and  distance. 
The  alternative  system  should  also  reduce  the  overall  cost  of  the  display  system.  With  new 
Commercial  Off  the  Shelf  (COTS)  and  Government  Off  the  Shelf  (GOTS)  technology,  it  is 
possible  to  provide  improved  imaging  at  a  reduced  cost.  The  desired  system  should  include  low- 
cost  collimated  projection  screens,  high  resolution,  full  color  laser  projectors,  and  high 
performance,  low-cost  image  generators  (Stermer,  2001). 
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2.3  INVESTMENT  ALTERNATIVES 


2.3.1  Process 

Utilizing  information  from  the  assessment  of  the  environment  and  the  gap  analysis, 
Human  Systems  IAC  then  adds  customer  input  and  the  opinions  of  technical  experts  to  identify 
potential  investment  alternatives.  That  is,  based  on  the  current  environment  assessment  and  the 
gap  analysis,  the  need  for  an  alternative  system  to  replace  the  status  quo  is  established.  Often, 
this  need  is  the  impetus  for  a  CBA. 

Alternatives  can  be  identified  in  several  manners.  In  general,  the  status  quo  (currently 
fielded  system)  is  one  of  the  alternatives  examined,  as  it  creates  a  baseline  for  comparison. 
Additional  alternatives  are  often  the  next  generation  of  technology  identified  in  the  research 
conducted.  That  is  the  case  with  this  study.  Human  Systems  IAC  analysts,  SMEs,  or  the 
customer  can  also  identify  further  choices.  For  the  DMT-A  CBA,  one  additional  alternative  was 
identified  to  compare  with  the  status  quo.  This  alternative  was  the  20/20  Immersive  Display 
System. 

2.3.2  Analysis 

2.3.2.1  Alternative  #1,  Status  Quo 

Described  fully  in  the  current  environment  assessment  (Section  2.1.2),  this  alternative  is 
the  system  currently  being  used.  It  consists  of  360-degree  visual  displays  with  reasonably 
effective,  but  limited  display  capabilities.  Some  of  these  limitations  include  brightness,  contrast, 
image  resolution  and  the  apparent  distance  of  the  projected  image  viewed  by  the  pilot.  Table  2 
describes  some  attributes  of  the  two  investment  alternatives  that  could  be  used  for  comparison. 

As  research  and  development  proceeds  and  more  detailed  information  becomes  available,  a  direct 
comparison  can  more  easily  be  made.  This  will  enhance  the  validity  of  future  efforts  and  make 
decisions  regarding  the  system  more  reliable. 


Table  2.  Comparison  of  Investment  Alternatives 


Display  Attributes 

Current  Display  System 

20/20  Immersive  Display 
System 

Field  of  view  (FOV) 

360  degrees 

360  degrees 

Brightness 

Acceptable 

Improved 

Image  resolution 

Acceptable 

Improved 

Projection  screen  transmissivity 

Acceptable 

Improved 

Contrast 

Acceptable 

Improved 
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Table  2.  Cont'd 


Training  Attributes 

Current  Display  System 

20/20  Immersive  Display 
System 

Effectiveness  against  multiple 
adversaries 

Effective 

More  effective 

Formation  flying 

Not  enough  detail  at  normal 
operational  distances 

Comparable  to  live-flight 

Target 

acquisition/identification  -  Air 
to  Air 

Not  enough  detail  at  normal 
operational  distances 

Improved  resolution  allows 
normal  cues  to  be  used  (e.g., 
direction  opponent  is  flying, 
flight  control  surfaces) 

Four  versus  X-number  of 
adversaries 

Limited  number  of  adversaries 

Unlimited  number  of 
adversaries 

Target 

acquisition/identification  -  Air 
to  Ground 

Overly  simplified  because  ground  is 
displayed  at  different  resolution 
than  target  imagery 

Integrated  ground  images  make 
scenarios  more  realistic. 

2.3.2.2  Alternative  #2, 20/20  Immersive  Display  System 

The  goal  of  the  20/20  Immersive  Visual  Display  System  for  DMT-A  is  to  develop, 
integrate,  and  demonstrate  a  high  fidelity  visual  display  system  for  flight  simulators.  This 
investment  alternative  is  intended  to  significantly  enhance  the  visual  system  for  fighter 
simulators  and  meet  Air  Combat  Command  (ACC)  goals  for  combat  operations  training  and 
mission  rehearsal.  This  will  be  accomplished  through  the  development  and  integration  of 
technologies  that  will  improve  the  resolution  of  the  simulated  visual  and  sensor  imagery  while 
reducing  the  overall  cost  of  the  image  generation  and  display  system.  The  target  visual  display 
system  will  meet  program  requirements  for  projections  screens,  projectors,  and  image  generators. 

The  resulting  visual  display  system  should  eliminate  the  need  for  target  projectors  that 
restrict  the  number  of  high  resolution  targets  that  can  be  displayed.  The  display  system  should 
also  improve  the  pilot’s  perception  of  an  object’s  size  and  distance  within  the  visual  scene.  The 
resulting  simulator  display  system  should  provide  complete  capability  for  those  missions  that 
require  visual  or  sensor  identification  of  mobile  ground  targets  within  a  high  definition, 
geospecific  terrain  and  cultural  database. 

These  improvements  will  be  the  result  of  the  addition  of  low-cost  collimated  projection 
screens  with  high  transmissivity,  and  high  resolution,  full  color  laser  projectors.  These 
projectors  would  display  over  10  times  the  number  of  pixels  currently  displayed  by  existing 
projectors.  High  performance,  low  cost  image  generators  based  on  commercial  technology  will 
provide  high  resolution  visual  and  sensor  imagery.  Major  program  deliverables  over  the  next 
several  years  include: 

•  High  definition  television  laser  projector, 

•  Large  scale,  collimated  screen, 

•  5K  x  4K  full  color  laser  projector, 

•  High  transmissivity,  collimated  screen, 

•  PC-based  image  generator  and  database  software, 

•  System  integration,  and 

•  Demonstration,  evaluation,  and  documentation. 
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2.4  COST  ESTIMATE 


2.4.1  Process 

Figure  2  summarizes  the  seven  main  activities  Human  Systems  IAC  uses  in  the  fourth 
step  of  the  CBA  process,  estimating  cost.  The  cost  of  each  of  the  alternatives  being  analyzed 
must  be  calculated  for  a  determined  period  of  time  (e.g.,  10-year  life  cycle)  to  enable  accurate 
cost  comparison  between  the  alternatives. 


■  Technical  Specialists 

•  Program  Analysts 

•  Cost  Analysts 
-LCCE  Schedule 

•  Finalize  cost 
proposal 

•  List  proposed 
deliverables 


•  Functional  Description  •  Integration  •  Similar  Systems  •  Cost  Estimating  -Estimating  Uncertainties  -Ground  Rules 

•  Mission  Requirements  -Schedule  •  Technology  Projections  Relationships  -Technical  Risk  •  Methodology 

•  Development  Scenario  •  Acquisition  Strategy  •  Cost /Technical  Data  Sources  Para  me  tries  -Schedule  Risk  •  Data  Sources 

•Technical  Characteristics  -Test  Plan  -Contractor Internal  Data  -Analogous Techniques -Program Risk  -Updates 

•  Program  Characteristics  •  MIL-STD-881B  •  Comparative  Analysis  •  Engineering  Methods  -Sensitivity  Analysis 

-Tech  Baseline  •  OSD  CAIG  O&S  Hdbk  •  Ops  &  Support  Estimate 

•  System  Description  •  Time  Phasing 

•  Inflation 

•  Budget  Profiles 

•  Logistics 

-  WBS/CES 


Figure  2.  Cost  Estimating  Activities 


Below  each  activity  is  a  listing  of  items  to  consider  and  sources  of  information  to 
reference  when  estimating  costs  of  the  alternatives.  Further  discussion  on  each  activity  follows. 

•  Team  Selection  and  Planning:  Team  selection  and  planning  activities  involve 
identification  of  the  program  team  members  who  will  be  assisting  with  the  estimating 
process  and  the  establishment  of  some  basic  information  related  to  source  selection 
(if  applicable). 

•  System  Characterization:  This  activity  is  very  important  to  the  success  or  failure  of 
the  entire  estimate.  During  this  activity,  the  analysts  begin  to  gain  an  understanding 
of  the  item  being  estimated.  This  includes  everything  from  basic  mission 
requirements  to  the  technical  characteristics  of  the  system. 

•  Ground  Rules  and  Assumptions:  During  this  activity,  the  cost  analyst  must  work 
closely  with  the  customer  to  establish  key  ground  rules  and  assumptions  for  the 
estimate.  This  is  where  the  estimate  begins  to  take  shape  in  terms  of  the  time 
phasing,  integration  requirements  and  cost  element  structure.  If  one  exists,  the 
program  Work  Breakdown  Structure  (WBS)  (MIL  HDBK  881,  1998)  should  be  used 


9 


as  a  baseline  to  formulate  a  cost  element  structure  (CES)  for  the  estimate.  The  CES 
defines  the  cost  estimate  elements  to  be  used,  while  providing  a  framework  from 
which  comparisons  between  the  alternative  solutions  can  be  made.  Table  2  below  is 
a  sample  generic  cost  element  structure. 


Table  3.  Sample  Generic  Cost  Element  Structure 


Development  and  Demonstration 

Production  and  Deployment 

Operation  and 
Support 

Program  Planning  and  Management 

Program  management 

Systems  Management 

Hardware 

Hardware 

Hardware  Maintenance 

Software 

Software 

Software  Maintenance 

Documentation 

Initial  Data  Requirements 

Site  Operations 

Training 

Initial  Training 

System  Test  and  Evaluation 

System  Integration  Test  &  Evaluation 

Logistics 

Support  Equipment 

Facilities 

Site  Activation 

Other 

Initial  Spares 

Industrial  Facilities 

Warranties 

Initial  Supplies 

Engineering  Changes 

General  Support 

•  Data  Collection  and  Methodology  Selection:  Once  the  cost  element  structure  is 
established,  the  analyst  begins  to  collect  data.  Again,  the  analyst  must  work  very 
closely  with  the  customer  to  ensure  the  best  available  information  is  used  and  all 
assumptions  are  accurate.  During  this  activity,  an  initial  determination  is  made  on 
which  cost  estimating  methodology  will  be  used.  Early  in  the  program,  estimates 
may  need  to  be  based  on  comparisons  to  similar  systems  (if  they  exist).  Later  in  the 
development  process,  the  estimate  may  be  able  to  be  based  on  known  detailed  design 
costs  or  the  cost  of  previous  buys  of  the  same  or  similar  equipment. 

•  Application  of  Estimating  Techniques:  Within  this  activity  there  are  three 
commonly  used  cost  estimating  approaches. 

o  Analogy  Approach:  The  analogy  approach  is  based  on  direct  comparison  with 
historical  information  of  similar  existing  activities,  systems,  or  components. 

This  technique  subjectively  compares  the  new  system  with  one  or  more  existing 
systems  for  which  there  is  accurate  cost  and  technical  data.  Adjustments  may  be 
made  to  account  for  complexity  differences  between  the  two  systems.  This 
method  is  usually  quick  and  inexpensive,  but  may  not  be  as  precise,  given  the 
subjective  nature  of  the  comparison. 

o  Parametric  Approach:  The  parametric  approach  is  based  on  costing  physical 
attributes  or  performance  characteristics  and  their  relationships  to  highly 
aggregated  component  costs.  Cost  estimating  relationships  (CER)  are  used  to 
predict  the  cost  of  an  item  or  effort  as  a  function  of  one  or  more  independent  or 
explanatory  variables  (e.g.,  physical  characteristics).  This  method  can  easily 
accommodate  changes  in  design,  performance  and  programmatic  characteristics 
(e.g.,  "what-if"  drills),  but  the  precision  of  this  approach  is  only  as  good  as  the 
source  database. 
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o  Engineering  Approach:  The  engineering  approach  is  a  "bottoms-up"  method  that 
requires  the  analyst  to  have  an  extensive  knowledge  of  the  system 
characteristics.  This  enables  him/her  to  construct  a  detailed  build  up  of 
estimates,  usually  by  specific  task  (e.g.,  labor  hours  x  labor  rates,  material  cost). 
Frequently,  actual  costs  available  to  this  level  of  detail  are  used  as  a  basis  to 
determine  first  unit  costs  that  can  then  be  used  in  the  application  of  learning 
curve  theory.  This  method  of  cost  estimating  is  the  most  detailed  of  all 
techniques  and  the  most  time  consuming. 

•  Uncertainty  and  Risk  Evaluation:  During  this  activity,  the  analyst  reviews  all  cost 
elements  for  potential  technical,  schedule,  and  program  risk.  Normally,  the  largest 
cost  drivers  are  carefully  examined  and  a  sensitivity  analysis  is  performed.  See 
sections  2.5  and  2.8  for  a  further  discussion  of  sensitivity  and  risk  analyses  as  they 
relate  to  the  overall  CBA  process. 

•  Cost  Estimate  Documentation:  The  final  activity  of  any  cost  estimate  is  to  ensure  the 
documentation  to  support  the  estimate  is  in  order.  At  a  minimum,  it  should  clearly 
outline  the  ground  rules  and  assumptions,  the  methodology  employed,  and  the  data 
sources  used. 

2.4.2  Discussion 

When  conducting  a  cost  analysis  for  DMT-A,  the  main  hardware  components  of  the 
20/20  Immersive  Display  system  are  already  known,  so  these  elements  would  serve  as  the 
starting  point  for  development  of  the  Cost  Element  Structure.  Items  such  as  the  high  definition 
laser  projector,  collimated  screen  and  PC-based  image  generator  would  all  be  separate  hardware 
elements  in  the  estimate.  By  working  closely  with  the  technical  experts  and  the  users  of  this 
proposed  system,  the  cost  analyst  will  gain  a  better  understanding  of  the  system  requirements  and 
should  therefore  be  able  to  determine  which  methodology  to  employ  to  obtain  the  most  accurate 
cost  estimate. 

2.5  SENSITIVITY  ANALYSIS 
2.5.1  Process 

The  sensitivity  analysis  illustrates  how  changes  in  key  assumptions  and  variables  may 
have  an  impact  on  the  total  cost  estimate.  These  changes  are  often  referred  to  as  "excursions." 
They  answer  the  question,  "What  elements,  if  changed,  would  influence  the  overall  cost  of  each 
alternative?"  The  result  of  such  an  excursion  assesses  the  magnitude  of  change  required  within 
key  cost  elements  sufficient  to  influence  the  outcome  of  the  cost  analysis.  While  many  different 
excursions  could  be  evaluated,  the  ones  most  likely  to  be  of  interest  to  the  reviewers  of  the 
analysis  are  selected. 

Even  though  every  effort  is  made  to  ensure  the  accuracy  of  cost  estimates,  the  very 
nature  of  forecasting  into  the  future  presents  a  certain  amount  of  risk  and  uncertainty.  As  long  as 
the  risk  is  identified,  it  can  be  managed  and  controlled.  To  account  for  the  uncertainty  and  the 
lack  of  precision  in  each  of  the  assumptions,  input  variable  distributions  (minimum,  most  likely, 
maximum)  can  be  estimated  for  key  cost  elements.  Once  the  cost  estimate  model  is  completed 
for  each  alternative  with  the  input  variable  distributions,  the  model  can  then  be  subjected  to  a 
Monte  Carlo  simulation. 

A  Monte  Carlo  simulation  calculates  numerous  scenarios  of  a  model  by  repeatedly 
picking  random  values  from  the  input  variable  distributions  for  each  "uncertain"  variable  and 
calculating  the  results.  The  results  of  Monte  Carlo  simulations  are  risk-adjusted  estimates  and 
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corresponding  statistical  estimate  distributions.  The  estimate  distributions  provide  the  decision¬ 
maker  with  a  range  of  possible  outcomes  and  bounds,  with  a  minimum  and  maximum  value.  The 
input  variable  distributions  and  cost  estimate  range  is  provided  with  each  alternative  analysis. 

While  sensitivity  analyses  can  occur  at  any  stage  of  a  CBA,  it  generally  makes  sense  to 
derive  an  unconstrained  material  solution  that  meets  all  mission  objectives  initially,  and  then 
begin  to  “back  off’  that  solution  in  the  interests  of  saving  money.  Care  must  be  taken,  however, 
not  to  impact  the  material  solution  to  such  an  extent  that  the  benefits  derived  from  that  solution 
are  significantly  altered  through  introduction  of  the  changes. 

2.5.2  Discussion 

When  conducting  a  sensitivity  analysis  on  the  visual  display  system  for  the  DMT-A 
program,  it  will  be  important  to  select  excursions  that  will  have  a  significant  impact  on  the  costs 
of  the  system.  For  instance,  the  more  complex  the  technologies  that  will  be  inserted,  the  more 
likely  it  will  be  that  budget  and  time  constraints  will  be  sensitive  to  change.  This  could  be  due  to 
the  reliability  of  components  or  the  stability  of  the  software  to  be  used.  Delays  resulting  from 
unreliable  components  could  shift  schedules  and  cause  unforeseen  and  unbudgeted  delays.  Thus, 
these  are  areas  that  will  likely  be  considered.  Additional  areas  of  concern  may  be  the  reliability 
of  cost  data  or  the  type  of  benefit  information  available  for  analysis.  However,  it  is  necessary  to 
carefully  examine  the  actual  cost  elements  to  determine  where  further  analysis  needs  to  occur. 

2.6  BENEFITS  CHARACTERIZATION  AND  VALIDATION 

2.6.1  Process 

In  addition  to  the  costs,  a  CBA  identifies  and  analyzes  the  benefits  of  each  of  the 
investment  alternatives.  In  many  respects,  this  step  is  a  logical  continuation  of  Step  3, 
identification  of  investment  alternatives.  Figure  3  illustrates  the  relationship  between  the  process 
used  to  define  investment  alternatives  and  the  process  for  identifying  benefits  associated  with 
those  alternatives.  The  key  to  this  relationship  is  a  thorough  understanding  and  documentation  of 
the  mission  requirements,  which  includes  goals  and  objectives  of  the  acquisition.  Ultimately,  the 
CBA  is  attempting  to  identify  how  to  measure  the  attainment  of  the  goals  and  objectives,  and 
then  "score"  each  alternative  on  the  extent  to  which  it  satisfies  those  goals  and  objectives. 
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•Identify  Functional  Requirement 
•MNS/ORD 

•Mission  Area  Roadmap 

•Define  Goals  and  Objectives 

•Key  Performance  Parameters 
•Acknowledge  Tolerable  levels  of  risk 


Identify  Alternatives 


•Define  Material  Solutions 

•Investigate  current  Inventory  of  Systems  and  Technologies 
•Investigate  feasibility  of  using  particular  designs/technologies 


•Decompose  goals  and  objectives  down  to  lowest  level 

•A  lowest  level  goal/objective  is  specifically  measurable 
•Include  ALL  lowest  level  goals/objectives 


Characterize  Benefits 


•Define  Criteria  for  scoring  alternatives 


Figure  3.  Relationship  Between  Identifying  Alternatives  and  Characterizing  Benefits 


It  is  important  that  all  significant  benefits,  whether  quantifiable  or  nonquantifiable,  be 
included  in  the  analysis.  Quantifiable  benefits  are  those  that  can  be  measured  or  assigned  a 
numeric  value,  such  as  dollars,  physical  count  of  tangible  items,  time,  revenue,  or  percentage 
change.  Dollar  valued  benefits  comprise  cost  reductions,  cost  avoidance,  and  productivity 
improvements.  Nonquantifiable  benefits  include  enhanced  information  security,  consistency 
and  compatibility  throughout  the  enterprise,  improved  quality,  enhancement  of  best  practices, 
adherence  to  statutory  and  regulatory  requirements,  and  enhanced  modernization. 

The  process  of  identifying  and  quantifying  benefits  is  often  the  most  difficult  part  of 
conducting  the  CBA.  It  is  important  to  follow  a  disciplined  approach  to  defining  benefits  to 
ensure  all  relevant  benefits  are  considered.  The  following  is  a  proposed  4-step  process  that  can 
be  used  to  benefit  identification  (Defense  Financial  Management  Reference  Guide,  2000): 

•  Identify  the  types  of  resources  flowing  into  the  system  and  identify  the  resulting 
benefits  flowing  out. 

•  Anticipate  what  is  important  (both  positively  and  negatively)  from  the  viewpoint  of 
each  person  affected  by  the  system  (stakeholders). 

•  Define  each  benefit  in  relation  to  the  alternatives.  All  benefits  included  must  be 
relevant,  clearly  and  distinctly  identifiable,  and  should  not  overlap  any  other 
measure. 

•  List  the  benefits  of  each  alternative,  both  quantifiable  and  nonquantifiable. 
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Typically,  some  benefits  may  be  characterized  in  financial  terms,  while  other  benefits 
may  not.  Financial  benefits  provide  the  most  efficient  means  of  determining  the  net  value  of  the 
investment  alternative  as  they  provide  a  net  gain  or  loss  of  dollars  based  on  the  impact  of  the 
benefit. 

Benefits  gained  from  the  use  of  human-centered  technologies  generally  fall  into  the 
category  of  nonquantifiable  benefits.  For  example,  improvements  in  the  reality  of  a  visual 
display  is  a  benefit,  but  difficult  to  quantify.  While  difficult  to  assess,  it  is  important  to  develop 
some  system  of  measurement  to  allow  direct  comparison  of  alternatives  for  each  benefit.  Several 
automated  tools  exist  for  benefits  evaluation.  Automated  evaluation  tools  attempt  to  minimize 
the  guesswork  in  decision-making.  The  underlying  framework  is  an  analytical  hierarchical 
process  used  to  organize  the  thoughts  and  intuition  of  key  stakeholders  in  a  logical  fashion.  This 
hierarchic  approach  allows  the  decision  maker  to  analyze  all  options  for  more  efficient  decision¬ 
making.  One  such  decision  support  software  tool  that  is  widely  used  in  both  industry  and 
Government  evaluations  is  Expert  Choice. 

2.6.2  Discussion 

When  conducting  a  benefits  analysis  for  DMT-A,  it  will  be  important  to  address  both  the 
quantitative  and  nonquantitative  benefits.  Some  potential  quantitative  benefits  that  may 
influence  the  net  value  would  be  cost  savings  related  to  sustainability  and  measurable  display 
improvements.  Some  potential  nonquantitative  benefits  are  listed  below. 

•  Compatibility  -  Will  current  or  new  display  system  be  compatible  with  existing  and 
future  upgrades  of  DMT-A  technology? 

•  Functionality  -  Will  display  system  operate  at  same  or  increased  rate?  Will  display 
system  exhibit  new  or  improved  functions  if  upgraded? 

•  Maintainability  -  Will  changes  in  system  influence  the  ease  and  efficiency  of  visual 
display  maintenance? 

•  Mission  effectiveness  -  Will  changes  in  the  display  system  have  an  influence  on 
mission  effectiveness?  Will  a  more  realistic  image  result  in  more  bombs  on  target? 

•  Morale  -  Will  changes  in  visual  display  system  have  an  influence  on  user  morale? 

•  Safety  -  Will  changes  in  the  visual  display  system  influence  safety  in  live  flight? 

Will  lessons  learned  in  simulation  result  in  fewer  casualties? 

•  Situational  awareness  -  Will  changes  in  the  display  system  have  an  influence  on 
operator  situational  awareness? 

•  Teamwork  -  Will  changes  in  the  visual  display  system  have  an  influence  on 
operators’  ability  to  work  as  a  team? 

•  Usability  -  Will  changes  in  the  visual  display  influence  the  ease  of  operator  use? 

Once  all  quantitative  and  nonquantitative  benefits  are  identified,  a  total  benefits  rating 
for  each  investment  alternative  can  be  developed.  A  useful  tool  for  this  is  Expert  Choice.  With 
Expert  Choice,  key  stakeholders  (e.g.,  engineers,  pilots,  program  managers)  are  gathered  and 
asked  to  form  judgments  about  the  benefits  of  each  alternative  based  on  information  available. 
The  result  is  a  global  benefits  score  that  can  be  used  to  determine  the  net  value  of  each 
alternative. 
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2.7  NET  VALUE  ANALYSIS 


2.7.1  Process 

The  net  value  analysis  is  process  of  combining  life  cycle  costs  developed  in  Step  4  with 
the  benefits  evaluation  conducted  in  Step  6.  The  alternative  with  the  greatest  benefit-to-cost  ratio 
is,  by  definition,  the  most  cost  effective.  However,  in  some  cases,  the  most  cost-effective 
alternative  may  not  be  the  best  solution.  All  aspects  of  the  alternatives  must  be  considered, 
compared  to  one  another,  and  an  informed  decision  must  be  made. 

There  are  a  variety  of  statistical  methods  currently  in  use  for  determining  the  "net  value" 
of  alternatives.  Most  financial  and  project  managers  are  using  one  of  the  following  three: 

•  Net  Present  Value  (NPV), 

•  Payback  Period,  or 

•  Cost  Benefit  Ratio  (CBR). 

The  following  sections  examine  these  valuation  methods  in  more  detail. 

2.7.1.1  Net  Present  Value  (NPV) 

Most  corporate  finance  textbooks  recommend  net  present  value  (NPV)  as  the  preferred 
criterion  for  making  choices  among  investment  alternatives.  In  general  terms,  NPV  is  defined  as 
the  difference  between  the  present  value  of  benefits  and  the  present  value  of  costs  (OMB 
Circular  A-94).  All  costs  and  benefits  are  adjusted  to  "present  value"  by  using  discount  factors  to 
account  for  the  time  value  of  money.  Mathematically,  this  definition  is  as  shown  below: 


NPV  =  PV(Total  Benefits)  -  PV(Total  Cost) 

One  drawback  with  this  method  is  that  all  benefits  associated  with  an  alternative  must  be 
able  to  be  quantified  in  cost  or  financial  terms  in  order  to  be  included  in  the  above  equation. 

This  is  often  not  possible,  especially  when  examining  benefits  in  the  area  of  human  centered 
technology  integration  (e.g.,  impact  on  morale,  increased  productivity,  decreased  safety 
incidents). 

2.7.1.2  Payback  Period 

Computing  the  amount  of  time  it  takes  for  a  project  to  pay  for  itself  (or  return  its  initial 
investment)  is  another  commonly  used  method  for  selecting  among  alternative  courses  of  action. 
Typically,  the  relevant  time  period  is  expressed  in  terms  of  the  number  of  years  it  takes  before  an 
investment  breaks  even.  When  using  this  technique,  discounted  cash  flows  are  not  used.  In  the 
simplest  of  cases,  the  benefits  (or  returns)  begin  predictably  at  the  completion  of  the  investment 
phase  and  occur  in  an  equal  amount  each  time  period.  In  reality  however,  benefits  begin 
accruing  prior  to  completion  of  the  investment  phase  and  do  not  occur  in  equal  annual  amounts. 
In  both  simple  and  complex  situations,  the  payback  period  in  years,  x,  can  be  found  in 
accordance  with  the  following  formula  (where  t  =  time  periods  in  years): 
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t-x 

^  (Operational  Savings  +  Mission  Savings)  =  (Initial  Investment) 

t= i 

As  with  the  net  present  value  method,  this  method  requires  all  benefits  considered  to  be 
expressed  in  financial  terms  in  order  to  be  included  in  the  equation. 

2.7.1.3  Cost  Benefit  Ratio  (CBR) 

Another  often-used  method  is  Cost  Benefit  Ratio(CBR).  Computation  of  this  metric  is 

simple: 


Benefits 

Cost 

This  method  provides  the  flexibility  for  benefits  to  be  assigned  a  "value"  and  therefore 
be  considered  without  assigning  them  a  price  tag.  Worth  noting  is  that  this  seeming  simplicity 
can  be  deceptive  due  to  the  fact  that  neither  the  composition  of  the  numerator  or  the  denominator 
has  been  universally  defined  or  accepted.  Section  2.6  provides  a  suggested  method  for 
evaluating  non-quantifiable  benefits.  In  addition,  in  order  for  this  ratio  to  work,  one  must 
accurately  consider  in  all  investment  alternatives,  costs  associated  with  maintaining  status  quo 
system(s)  until  the  preferred  alternative  is  fully  fielded  and  any  overlapping  "systems"  are 
abandoned  or  "turned  off".  The  trick  to  making  this  ratio  work  is  the  development  of  life  cycle 
cost  estimates  (LCCEs)  for  the  Status  Quo  and  any  alternatives  to  it  that  fully  represent  the 
systems-related  costs  of  supporting  a  functional  mission.  Presuming  this  has  been  done,  then  a 
meaningful  CBR  can  be  computed. 

2.7.2  Discussion 

When  conducting  a  net  value  analysis  for  DMT-A,  the  choice  of  valuation  method  will 
be  key.  The  valuation  method  used  in  any  net  value  analysis  is  heavily  dependent  on  the 
program  manager's  priorities.  For  example,  if  the  analysis  is  being  conducted  to  determine  the 
most  expedient  solution,  then  payback  period  will  be  the  preferred  method  of  valuation.  If  the 
motivation  is  determining  which  solution  provides  the  greatest  amount  of  benefit  for  the  funds 
invested,  then  CBR  is  the  most  appropriate  method.  The  bottom  line  is  that  there  are  several 
acceptable  methods  of  calculating  the  net  value  of  competing  alternatives.  Selecting  the  right 
one  for  DMT-A  will  require  additional  information  and  consideration  by  the  sponsor  to  make  a 
recommendation. 

2.8  RISK  ANALYSIS 

2.8.1  Process 

In  the  final  step  of  a  cost  benefit  analysis,  both  costs  and  benefits  are  investigated  from  a 
risk  perspective.  The  objective  of  this  step  is  to  isolate  areas  of  the  alternatives  where 
uncertainties  exist  in  the  analyses.  Once  completed,  program  managers  then  possess  the  added 
foresight  to  focus  their  attention  on  those  risky  areas  when  making  programmatic  decisions.  In 
general,  the  more  complex  the  alternative,  the  more  likely  it  is  that  changes  in  schedule/cost,  or 
benefit  assumptions  will  result.  For  example,  if  certain  cost  assumptions  or  benefit  assessments 
change  dramatically,  the  overall  findings  will  likely  change  as  well. 
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There  are  various  ways  to  categorize  risks  that  affect  investment  alternatives.  Provided 
below  are  definitions  for  five  types  of  risk.  Each  program  must  separately  determine  the 
categories  of  risk  to  evaluate. 

•  Technical  Risk:  Technical  risk  is  defined  as  uncertainty  in  the  system  performance 
or  “benefits.”  Technical  risk  may  result  from  an  immature  technology,  use  of  a 
lower-reliability  component,  degree  to  which  products  employ  the  latest  standards  in 
technology  and  design,  availability  of  skilled  resources  to  support  the  product,  and 
then  degree  of  tailoring  required.  Technical  risk  can  be  reflected  in  increased  costs 
(to  fix  the  technical  problem)  and  lower  overall  system  benefits. 

•  Integration  Complexity  Risk:  This  category  includes  risks  associated  with  the 
number  of  data  dependencies,  the  number  of  actual  interfaces  between  this  module 
and  other  modules,  and  the  technical  issues  involved  regarding  programming  and 
application  solutions. 

•  Market  Risk:  This  category  includes  risks  associated  with  the  stability  of  vendors 
and  their  software  and  related  tools  and  services  within  the  market  (in  this  case 
federal  HR  commercial  off-the-shelf  [COTS]  product  market).  Market  risk  may 
increase  or  decrease  depending  on  such  factors  as  the  number  of  vendors  or  products 
within  the  market,  the  degree  to  which  specific  products  are  tested  and  implemented 
in  a  production  environment  similar  to  the  intended  use  of  the  weapons  system  under 
consideration,  and  implementation. 

•  Cost  Risk:  Cost  risk  is  defined  as  uncertainty  resulting  from  the  use  of  a  particular 
cost  estimating  methodology.  As  discussed  in  Section  2.4,  each  cost  estimating 
methodology  brings  with  it  different  strengths  and  weaknesses.  It  is  during  the  risk 
analysis  that  these  strengths  and  weaknesses  are  examined  further  for  impact  to  the 
overall  program. 

•  Implementation/Project  Risk:  This  category  includes  risks  that  the  module 
implementation  will  be  successful  and  run  according  to  planned  schedule.  Schedule 
risk  is  defined  as  uncertainty  in  the  project  completion  or  fielding  schedule,  and  the 
subsequent  impact  on  costs  and  level  of  benefits.  A  stretched-out  schedule  may 
increase  costs  due  to  extended  level-of-effort  funding  requirements,  and  result  in 
delivery  of  systems  too  late  to  have  the  desired  effect  (reduced  benefits).  This 
category  also  addresses  factors  such  as  the  thoroughness  of  project  approach  and 
plan,  the  degree  to  which  plans  incorporate  risk  mitigation  techniques,  and  the 
impact  of  not  meeting  or  adjusting  the  project’s  anticipated  timeline. 

2.8.2  Discussion 

When  conducting  a  risk  analysis  on  the  visual  display  system  of  DMT-A,  several  factors 
will  need  to  be  considered.  Some  important  factors  that  will  influence  the  uncertainties  of  the 
estimation  are  the  technical  maturity  of  the  display  products  to  be  integrated,  the  ease  of 
integrating  new  visual  displays  with  existing  technology,  the  stability  of  vendors,  and  the 
likelihood  that  implementation  will  take  place  on  time  and  within  budget.  Clearly,  these  are 
areas  that  can  only  be  adequately  addressed  once  all  others  steps  of  the  CBA  are  complete. 
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3.  CONCLUSIONS  AND  RECOMMENDATIONS 


Human  Systems  IAC  was  asked  to  perform  the  preliminary  steps  associated  with 
assessing  the  costs  and  benefits  of  two  alternatives  for  the  visual  display  system  of  the  distributed 
mission  training  program.  This  report  also  outlines  the  remaining  five  steps  that  would  need  to 
be  performed  in  order  to  complete  a  CBA. 

Based  on  the  information  available,  the  DMT-A  program  is  an  effective  tool  for  training 
pilots  to  work  as  a  team.  Based  on  the  gap  analysis,  visual  display  resolution  requirements  are 
not  being  met  completely  and  the  visual  display  system  may  need  to  be  upgraded.  A  potential 
upgrade  to  the  status  quo  visual  display  system  is  the  20/20  Immersive  Visual  Display  System. 

As  additional  details  regarding  the  alternatives  become  available,  the  new  information  will 
improve  the  reliability  of  future  analysis  efforts.  The  next  steps  to  determining  whether  a  new 
display  system  is  warranted  would  be  to  assess  the  costs  and  benefits  associated  with  each 
display  system.  In  addition,  an  evaluation  of  sensitivity  and  risk  would  need  to  be  accomplished. 
Following  that,  the  net  value  analysis  of  the  investment  alternatives  would  determine  which  was 
the  preferred  alternative. 
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APPENDIX  A.  LITERATURE  SEARCH  STRATEGY 


Distributed  Mission  Training  Cost  Benefit  Analysis 
Literature  Search  Strategy 

For:  Air  Force  Research  Laboratory 

Wright-Patterson  Air  Force  Base,  OH 

Background: 

Human  Systems  IAC  is  conducting  the  first  3  phases  of  the  cost  benefit  analysis  process 
associated  with  Distributed  Mission  Training  (DMT)  at  a  global  level.  These  phases  are  1) 
Assess  Current  Environment,  2)  Perform  Gap  Analysis,  and  3)  Identify  Investment  Alternatives. 
DMT  technology  allows  multiple  players  at  multiple  sites  engage  in  training  scenarios  ranging 
from  individual  and  team/battle-staff  participation  to  full  theater-level  battles.  Additional 
technologies  (e.g.,  constructive  forces)  provide  for  nearly  unlimited  training  opportunities  for 
joint  and  combined  forces.  DMT  dramatically  improves  the  quality  and  quantity  of  warfighter 
training. 

The  specific  goal  of  this  literature  search  is  to  review  recent  literature  on  DMT  to  help 
Human  Systems  IAC  identify  and  understand  the  current  level  of  technology  and  research  and 
development  (R&D),  specifically  in  government  training  applications.  This  review  should  also 
help  identify  the  available  alternatives  and  what  the  potential  payoffs  are  for  continuing  R&D  in 
the  area  of  DMT.  This  information  will  then  be  evaluated  with  the  cost  data  to  determine  the 
alternative  technology  with  the  greatest  return  on  investment. 

The  results  of  the  literature  search  strategy  will  be  used  to  derive  cost  and  benefits  as 
appropriate.  The  results  are  especially  important  in  capturing  the  "value"  of  benefits  in  order  to 
quantify  them  in  our  final  analysis. 


Search  Terms: 

See  attached  table  of  terms. 

Key  Authors: 

Jesse  Orlansky 
Henry  L.  Taylor 


Possible  Databases:  (Final  list  to  be  determined  based  on  customer  and  expert  searcher 
recommendation  s) 

Aerospace  Database 
DTIC 

ISI  Science  Citation  Index 

NASA  Recon 

NTIS 

PsychINFO 
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Example  Articles: 

Dwyer,  D.,  Oser,  R.,  &  Fowlkes,  J.  (1995).  A  case  study  of  distributed  training  and  training 

performance  (pp.  1316-1320).  In  Proceedings  of  the  Human  Factors  and  Ergonomics  Society 
39th  Annual  Meeting,  San  Diego,  CA,  October  9-13.  Santa  Monica,  CA:  Human  Factors  and 
Ergonomics  Society. 

Jackson,  P.  M.  (1996).  Design  considerations  for  interoperability  in  mission  training  systems  (pp. 
193-198).  In  S.  Sestito,  P.  Beckett,  G.  Tudor,  &  R.  Triggs  (Eds.),  SimTec  Proceedings. 
Melbourne,  Australia:  SimTechT  Organising  Committee. 

Lane,  N.  E.  &  Alluisi,  E.  A.  (1992).  Fidelity  and  validity  in  distributed  interactive  simulation: 

Questions  and  answers  (DTIC  No.  AD-A260971).  Alexandria,  VA:  Institute  for  Defense 
Analyses. 

Mudd,  D„  Pierce,  B.,  Gasson,  R.,  Greschke,  D.,  &  Svendson,  J.  (1998).  Canadian  Forces  distributed 
mission  training  feasibility  study  (CF-18  Multitask  Trainer)  (Report  No.  AFRL-HE-AZ-TR- 
1998-0084-V-l).  Mesa,  AZ:  Raytheon  Training  and  Services. 

Orlansky,  J.,  Taylor,  J.  Levine,  D,  &  Honig,  J.  (1997).  The  cost  and  effectiveness  of  the  Multi- 
Service  Distributed  Training  Testbed  (MDT2)  for  training  close  air  support  (Report  No. 
IDA/HQ-96-004084).  Alexandria,  VA:  Institute  for  Defense  Analyses. 
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Goal:  To  acquire  any  information  related  to  simulation  and  training. 


APPENDIX  B.  COMMON  ECONOMIC  DEFINITIONS 


Alternative — An  approach  or  program  that  is  another  possible  way  of  fulfilling  an  objective, 
mission,  or  requirement.  The  status  quo,  or  an  upgrade  to  the  status  quo,  is  usually  an  alternative 
to  a  proposed  course  of  action. 

Analytic  Hierarchy  Process  —The  Analytic  Hierarchy  Process  (AHP)  is  a  powerful  and 
flexible  decision-making  process  used  to  assist  in  setting  priorities  and  making  the  best  decision 
when  both  qualitative  and  quantitative  aspects  of  a  decision  need  to  be  considered.  AHP  reduces 
complex  decisions  to  a  series  of  one-on-one  comparisons,  then  synthesizes  the  results.  AHP  was 
developed  more  than  20  years  ago  by  Dr.  Thomas  Saaty,  and  continues  to  be  one  of  the  most 
highly  regarded  and  widely  used  decision-making  theories. 

Analysis  of  Alternatives — An  Analysis  of  Alternatives  AO  A)  evaluates  the  costs  and  benefits 
(i.e.,  the  operational  effectiveness  or  military  utility)  of  alternative  courses  of  action  to  meet 
recognized  defense  needs. 

Base  Year — A  point  of  reference  that  represents  a  fixed  price  level,  usually  defined  as  the  fiscal 
year  when  a  program  was  initially  funded.  Expressing  program  costs  in  a  specified  base  year  is 
the  same  as  expressing  those  costs  in  constant  year  dollars  of  the  same  year. 

Base- Year  Dollar — Reflects  the  dollar's  value  at  the  midpoint  (April  1)  of  a  given  base  year  and 
implies  this  value  throughout  the  base  year.  When  cost  estimates  are  in  base-year  dollars,  the 
purchasing  power  of  the  dollar  remains  the  same  over  the  estimated  time  period. 

Benefit — An  objective  measure  of  an  alternative's  value.  When  a  dollar  value  cannot  be  placed 
on  comparable  program  or  project  benefits,  other  objective  measures  may  be  available  and  useful 
for  comparing  alternatives. 

Benefits  Analysis — Analysis  to  identify,  measure  and  evaluate  the  benefits  for  each  proposed 
alternative. 

Benefit  to  Cost  Ratio — An  alternative  way  to  express  "Cost  Benefit  Ratio".  See  Cost  Benefit 
Ratio. 

Commercial-Off-The-Shelf — Hardware  or  software  products  developed,  tested,  and  sold  by 
commercial  companies  to  the  general  public. 

Constant  Year  Dollar — Reflects  the  value  or  purchasing  power  of  a  dollar  in  any  specific  year, 
which  may  or  may  not  be  the  base  year.  Constant  year  dollars  do  not  contain  any  adjustments  for 
inflation  that  has  occurred  or  is  forecasted  to  occur  outside  the  base  year.  Constant  year  dollars 
are  also  not  influenced  by  outlay  profiles  (expenditure  patterns). 

Cost  Model — A  compilation  of  cost  estimating  logic  that  aggregates  cost  estimating  details  into 
a  total  cost  estimate;  An  ordered  arrangement  of  data,  assumptions,  and  equations  that  permits 
translation  of  physical  resources  or  characteristics  into  costs.  Cost  models  generally: 

(a)  Consist  of  a  set  of  equations,  logic,  programs,  and  input  formats  to  specify  the  problem 
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(b)  Allow  for  the  input  of  program  information,  including  both  system  description  data  and 
estimating  relationships 

(c)  Produce  an  output  format. 

Cost  and  Operational  Effectiveness  Analysis — The  COEA  provides  an  analytical  basis  to 
support  milestone  decision  reviews  by  allowing  comparison  of  alternative  solutions  based  on  cost 
and  performance. 

Cost  Benefit  Analysis — A  systematic  approach  to  the  problem  of  choosing  how  to  use  scarce 
resources.  It  reveals  the  monetary  value  of  the  costs  and  benefits  associated  with  all  alternatives 
under  consideration,  and  provides  as  accurate  and  complete  a  picture  as  possible  of  nonmonetary 
costs  and  benefits. 

Cost  Benefit  Ratio — The  mathematical  expression  of  the  relative  cost  per  unit  of  benefit 
obtained;  lower  ratios  are  preferred  as  this  indicates  that  more  benefit  is  gained  for  each  dollar 
invested  (Total  Cost  /  Total  Benefits). 

Cost  Element — An  identifiable  function,  or  a  common  group  of  functions,  which  have  been 
established  as  a  separate  entity  for  the  purpose  of  estimating,  collecting,  controlling,  and 
reporting  costs. 

Cost  Element  Structure — The  documented  configuration  of  common  functions  which  are  used 
to  systematically  estimate,  collect,  control  and  report  program  costs. 

Cost  Estimating  Relationship — A  mathematical  expression  defining  cost  as  the  dependent 
variable  to  one  or  more  independent  cost  driving  variables. 

Discount  Factor — The  factor  that  translates  expected  benefits  or  costs  in  any  given  future  year 
into  present  value  terms.  The  discount  factor  is  equal  to  1/(1  +  i)t 

where  i  is  the  interest  rate  and  t  is  the  number  of  years  from  the  date  of  initiation  for  the  program 
or  policy  until  the  given  future  year. 

Discounted  Cash  Flow — Periodic  money  stream(s)  resulting  from  economic  decisions  that  are 
adjusted  for  the  time  value  of  money  (interest  rate).  Discounting  cash  flows  allows  for  common 
comparison  among  competing  alternatives  that  offer  returns  over  different  periods  of  time. 

Economic  Life — The  period  of  time  during  which  the  benefits  from  a  project  may  reasonably  be 
expected  to  accrue. 

Government-Off-The-Shelf — An  item  of  hardware  or  software  that  has  been  produced  by  or  for 
the  government  and  is  available  for  reuse;  Products  for  which  the  government  owns  the  data 
rights,  that  are  authorized  to  be  transferred  to  other  DoD  or  U.S.  Government  customers,  and  that 
require  no  unique  modifications  or  maintenance  over  the 

Internal  Rate  of  Return — The  present  value  of  the  net  cash  flow  stream  expressed  as  an  annual 
percentage  growth  rate  of  the  initial  investment  (i.e.  the  annual  compound  interest  rate  required 
to  produce  the  same  result  as  the  evaluated  investment). 
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Investment  Analysis — The  process  of  examining  the  feasibility  of  a  particular  course  of  action 
based  on  the  costs  to  pursue  it  and  resulting  benefits  (savings)  from  doing  so. 

Investment  Costs — Those  program  costs  required  beyond  the  development  phase  to  introduce 
into  operational  use  a  new  capability;  to  procure  initial,  additional,  or  replacement  equipment  for 
operational  forces;  or  to  provide  for  major  modifications  of  an  existing  capability.  They  exclude 
research,  development,,  test  and  evaluation,  military  personnel,  and  operation  and  maintenance 
appropriation  costs. 

Life-Cycle  Cost — The  total  cost  to  the  government  for  a  system  over  its  useful  life,  including  the 
cost  of  development,  procurement,  operation,  support,  and  disposal 

Measures  of  Effectiveness — Operational  capabilities  stated  in  terms  of  engagement  or 
battlefield  outcomes.  In  the  modeling  process,  they  are  used  to  assist  in  discriminating  among 
competing  alternatives.  They  show  how  the  alternatives  compare  in  meeting  functional 
objectives  and  mission  needs. 

Measures  of  Performance — MOPs  are  technical  data  elements  supporting  a  MOE. 

Mission  Life — The  time  period  of  program  use  or  operation. 

Mission  Needs  Statement — A  requirements  document  that  outlines  a  deficiency  based  on 
operational  capability,  rather  than  system  specifics.  The  MNS  identifies  the  need  to  establish  a 
new  operational  capability,  improve  existing  capabilities,  or  exploit  an  opportunity  that  cannot  be 
satisfied  with  non-material  solutions. 

Net  Present  Value — The  present  value  of  all  revenue  (savings)  streams  less  the  present  value  of 
all  cost  (investment)  streams. 

Office  of  Management  and  Budget — Office  that  assists  the  President  in  overseeing  the 
preparation  of  the  Federal  budget  and  in  supervising  its  administration  of  Federal  agencies.  The 
OMB  also  oversees  and  coordinates  the  Administration's  procurement,  financial  management, 
information  and  regulatory  policies. 

OMB  Circulars  and  Bulletins — Publications  used  by  the  Executive  office  of  the  President  to 
exercise  managerial  and  policy  direction  over  Federal  agencies.  Circulars  and  Bulletins 
generally  promote  government  effectiveness  by  providing  uniform  guidance  to  agencies. 

Operational  Requirements  Document — A  document  which  details  the  users  objectives  and 
minimum  acceptable  requirements  for  operational  performance  of  a  proposed  concept  or  system. 

Present  Value — Monies  accruing  in  the  future  that  have  been  discounted  to  account  for  the  fact 
that  those  monies  will  be  worth  less  in  the  future  than  they  are  today. 

Recurring  Costs — Expenses  for  personnel,  material  consumed  in  use,  overhead  support 
services,  and  other  items  incurred  on  a  repeating  basis. 

Return  on  Investment — Net  income  divided  by  investments. 
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Risk — The  likelihood  that  some  assumption  or  estimate  is  wrong.  Sometimes  used 
synonymously  with  "uncertainty,"  though  uncertainty  can  be  described  by  a  probability 
distribution. 

Savings  to  Investment  Ratio — The  ratio  of  the  present  value  of  a  savings  stream  with  respect  to 
the  present  value  of  the  cost  of  making  the  investment. 

Sensitivity  Analysis — Examination  of  the  effects  obtained  by  changing  the  direction  and 
magnitude  of  assumptions  embodied  in  an  analysis  or  key  variables  or  factors  in  an  analysis. 

Sunk  Cost — The  sum  of  past  expenditures  or  irrevocably  committed  funds  related  to  a  project. 
Such  costs  are  generally  not  relevant  to  decision  making  as  they  reflect  previous  rather  than 
present  choices. 

Work  Breakdown  Structure — A  product-oriented  family  tree,  composed  of  hardware, 
software,  services,  and  other  work  tasks,  which  results  from  project  engineering  effort  during  the 
development  and  production  of  a  defense  materiel  item,  and  which  completely  defines  the  project 
or  program. 
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About  Human  Systems  IAC 

The  Human  Systems  Information  Analysis  Center  (Human  Systems  IAC,  HSLAC)  is  the  gateway 
to  worldwide  sources  of  up-to-date  human  factors  and  ergonomics  information  and  technologies 
for  designers,  engineers,  researchers,  and  human  factors  specialists.  Human  Systems  IAC 
provides  a  variety  of  products  and  services  to  government,  industry,  and  academia  while 
promoting  the  use  of  human  factors  and  ergonomics  in  the  design  of  human-operated  equipment 
and  systems. 

Human  Systems  IAC’s  primary  objective  is  to  acquire,  analyze,  and  disseminate  timely 
information  on  human  factors  and  ergonomics.  In  addition  to  providing  free  basic  searches, 
Human  Systems  IAC  performs  other  services  on  a  cost-recovery  basis: 

•  Distribute  human  factors  and  ergonomics  technologies  and  publications 

•  Perform  customized  bibliographic  searches  and  literature  reviews 

•  Prepare  state-of-the-art  reports  and  critical  review 

•  Conduct  specialized  analyses  and  evaluations 

•  Organize  and  conduct  workshops  and  conferences 

Human  Systems  IAC  is  a  Department  of  Defense  Information  Analysis  Center  sponsored  by  the 
Defense  Technical  Information  Center.  It  is  technically  managed  by  the  Air  Force  Research 
Laboratory  Human  Effectiveness  Directorate  and  operated  by  Booz  Allen  &  Hamilton  Inc. 


Published  by: 

The  Human  Systems  Information  Analysis  Center 
HSIAC) 

ARFL/HEC/HSIAC 

2261  Monahan  Way,  Bldg.  196 

Wright-Patterson  AFB,  OH  45433-7022 

Phone:  (937)255-4842  DSN:  785-4842 

Fax:  (937)255-4823  DSN:  785-4823 

E-mail:  tom.metzler@wpafb.af.mil 

WWW:  http  ://iac.dtic.mil/hsiac./ 
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